Single Method Surveys and the Ridiculously High Cost of Mixed Designs

class: center, middle, inverse, title-slide

.title[
# Single Method Surveys and the Ridiculously High Cost of Mixed Designs
]
.subtitle[
## EDP 618 Week 4
]
.author[
### Dr. Abhik Roy
]

---

section {
    height: 600px;
    width: 60%;
    margin: auto;
    border-radius: 21px;
    background-color: #212121;
}

.remark-slide-container {
background: #212121;
}

.hljs-github .hljs {
    background: transparent;
    color: #b2dfdb;
}

.hljs-github .hljs-keyword {
    color: #64b5f6;
}

.hljs-github .hljs-literal {
    color: #64b5f6;
}

.hljs-github .hljs-number {
    color: #64b5f6;
}

.hljs-github .hljs-string {
    color: #b7b3ef;
}

.hljs-github .hljs {
    background: transparent;
    color: #b2dfdb;
}

.hljs-github .hljs-keyword {
    color: #64b5f6;
}

.hljs-github .hljs-literal {
    color: #64b5f6;
}

.hljs-github .hljs-number {
    color: #64b5f6;
}

.hljs-github .hljs-string {
    color: #b7b3ef;
}

section p {
    text-align: center;
    font-size: 30px;
    background-color: #212121;
    border-radius: 21px;
    font-family: Roboto Condensed;
    font-style: bold;
    padding: 12px;
    color: #bff4ee;
    margin: auto;
}

#center {
text-align: center;
}

#right {
  text-align: right;
}

.center p {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

.tab {
    display: inline-block;
    margin-left: 40px;
}

.obr
{
 display:block;
 margin-top:-15px;
}
</style>

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 128px;
z-index: 0;
background-image: url(/Users/skynet/Documents/WVU/Teaching/GitHub.nosync/edp618/static/img/course_hex.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
 let tries = 0
 function addLogo () {
 if (typeof slideshow === 'undefined') {
 tries += 1
 if (tries < 10) {
 setTimeout(addLogo, 100)
 }
 } else {
 document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
 .forEach(function (slide) {
 const logo = document.createElement('a')
 logo.classList = 'xaringan-extra-logo'
 logo.href = 'https://edp618.asocialdatascientist.com'
 slide.appendChild(logo)
 })
 }
 }
 document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

---

class: highlight-last-item
layout: true

---

# Part I: Survey Design and Crafting Items

---

##  The Tailored Design Method (TDM)

- Premised on social exchange perspective on human behavior - *You do something for me, and I do something for you*

- Assumes that the likelihood of responding is greater when the expected rewards outweigh the anticipated costs - *Increased benefits with decreased costs*

---

## Approaches to Increasing Benefits

- Provide information about the survey

- Ask for help or advice

- Show positive regard

- Say thank you!

- Support group values

- Give tangible rewards

- Make the questionnaire interesting

- Provide social validation

- Inform people that opportunities to respond are limited

---

## Approaches to Decreasing Costs

- Avoid subordinating language

- Emphasize similarity to other requests or tasks to which a person has already responded

- Make it convenient to respond

- Make the questionnaire short and easy to complete

- Minimize requests for personal or sensitive information

---

##  Establishing Trust

- Provide information about the survey

- Ask for help or advise

- Show positive regard

- Say thank you!

---

##  Choosing Words and Forming Question

+ Ask one question at a time

+ Be sure the question specifies the response task

+ Make sure the question applies to the respondent

+ Make sure the question is technically accurate

+ Make sure *yes* means **yes** and *no* means **no**

+ Use complete sentences with simple sentence structures

+ Use as few words as possible to pose the question

+ Use specific and concrete words to specify the concepts clearly

+ Use simple and familiar words

---

## Visual Presentation of Survey Questions (1/2)

+ Choose line spacing, font, and text size to ensure the legibility of the text

+ Integrate special instructions into the question where they will be used rather than including them as freestanding entities

+ Make sure words and visual elements that make up the question send consistent messages

+ Organize each question in a way that minimizes the need to reread portions in order to comprehend the response task

+ Separate optional or occasionally needed instructions from the question stem by font or symbol variation

---

## Visual Presentation of Survey Questions (2/2)

+ Use darker and/or larger print for the question and lighter and/or smaller print for answer choices and answer spaces

+ Use spacing to create subgrouping within a question

+ Use visual design properties to emphasize elements that are important to the respondent and to de-emphasize those that are not

+ Visually standardize all answer spaces or response options

---

##  First Things First: Avoid Double-Barreled Questions!

.pull-left[
<img src="img/image2a.png" width="75%" height="75%"/>
]

.pull-left[
<img src="img/image2b.png" width="75%" height="75%"/>
]

---

##  Double-Barreled Questions – What’s the problem?

<hr style="width:100px; margin: auto;" />
 
 
<center>
Respondents don’t know what you are asking for
</center>

<center>
If the answer is Yes, then are the respondents saying it for the burger, fries, or both?
</center>

<center>
This can lead to massive bias, unreliable or unusable results
</center>

---

## Structures

**The following slides consist of examples of common structures of survey items.**
 
***This is by no means comprehensive!***

---

###  Nominal Type Questions

.footnote[
*More about [nominal variables](https://statsandr.com/blog/variable-types-and-examples/##nominal)*
]

---

###  Dichotomous Nominal Type Questions

---

###  Ordinal Type Questions

.footnote[
*More about [ordinal variables](https://statsandr.com/blog/variable-types-and-examples/##ordinal)*
]

---

### More About Ordinal Type Questions

---

##### Ordinal or Nominal?

.pull-left[
<img src="img/image8a.png" width="95%" height="95%"/>
]

.pull-right[
<img src="img/image8b.png" width="95%" height="95%"/>
]

---

### Semantic Differentials

.pull-left[
<img src="img/image9a.png" width="65%" height="65%"/>
]

.pull-right[
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp; <img src="img/image9b.png" width="55%" height="55%"/>
]

---

### Visual Analogs

---

### Guttman Scaling

---

### Filters/Skip Patterns

Comes in many names:

> Skip logic

> Conditional Branching

> Conditional Logic

.pull-right[
<img src="img/image13.png" width="80%" height="80%" />
]

---

## Closed Ended

---

## General Structural Guidelines

- State both positive and negative sides in the question stem when asking either/or types of questions

- Develop lists of answer categories that include all reasonable possible answers

- Develop lists of answer categories that are mutually exclusive

- Maintain spacing between answer categories that is consistent with measurement intent

---

### Positive and Negative Sides in Question Stem

.pull-left[
<img src="img/image14a.png" width="57%" height="57%" />
]

.pull-right[
<img src="img/image14c.png" width="80%" height="80%" />
]

.pull-left[
<img src="img/image14b.png" width="70%" height="70%" />
]

.pull-right[
<img src="img/image14d.png" width="80%" height="80%" />
]

---

### Exhaustive and Mutually Exclusive Questions

.pull-left[
<center>
<img src="img/image15.png" width="97%" height="97%" />
</center>
]

.pull-right[
<center>
<img src="img/image16.png" width="97%" height="97%" />
</center>
]

---

### Spacing Response Options Evenly

---

## Closed-Ended Questions: Nominal Scales

- Ask respondents to rank only a few items at once rather than a long list

- Avoid bias from unequal comparisons

- Randomize response options if there is concern about order effects

- Use forced-choice questions rather than check-all-that-apply questions

- Consider using differently shaped answer spaces (circles and squares if possible) to help respondents distinguish between single- and multiple-answer questions

---

### Unordered

.pull-right[
<img src="img/image18b.png" width="100%" height="100%" />
]

---

### Unordered

---

### Comparisons (1/2)

---

### Comparisons (2/2)

---

### Check-all-that-Apply versus Forced-Choice

.pull-left[
<center>
<img src="img/image20a.png" width="95%" height="95%" />
</center>
]

.pull-right[
<center>
<img src="img/image20b.png" width="95%" height="95%" />
</center>
]

---

### Distinguishing Between Single-and Multiple-Answer Questions

.pull-left[
<center>
<img src="img/image21a.png" width="100%" height="100%" />
</center>
]

.pull-right[
<center>
<img src="img/image21b.png" width="100%" height="100%" />
</center>
]

---

## Closed-Ended Questions: Ordinal Scales

- Align response options vertically in one column or horizontally in one row and strive for equal distance between categories

- Carefully evaluate the use of numeric labels and their impact on measurement

- Choose an appropriate scale length—in general, limit scales to four or five categories

- Choose direct or construct-specific labels to improve cognition

- Consider how verbally labeling and visually displaying all response categories may influence answers

- Place non-substantive options at the end of the scale and separate them from substantive options

- Provide scales that approximate the actual distribution of the characteristic in the population

- Provide balances scales where categories are relatively equal distances apart conceptually

---

### Scalar Questions (1/2)

---

### Scalar Questions (2/2)

.pull-left[
<center>
<img src="img/image23a.png" width="75%" height="75%" />
</center>
]

.pull-right[
<center>
<img src="img/image23b.png" width="75%" height="75%" />
</center>
]

</center>
.footnote[
*Note: Here we're referring to balanced scales as having choices that are equidistant from each other*
]

---

### Aligning the Conceptual and Visual Midpoints

**This is by far one of the most important criteria to check off when considering survey aesthetics!**

---

---

## Open Ended

---

### Open-Ended Numerical

- Ask for the specific unit desired in the question stem

- Provide answer spaces that are sized appropriately for the response task

- Provide units labels with the answer spaces

---

### Numeric

---

### Open-Ended Lists

- Design the answer spaces to support the number and type of responses desired

- Provide labels with answer spaces to reinforce the type of response requested

- Specify the number and type of responses desired in the question

---

### Lists

--
<center>
&nbsp; <img src="img/image26a.png" width="19%" height="19%" />
</center>

---

## Open-Ended Explanatory

- Consider programming probes to open-ended responses in internet surveys

- Provide adequate space for respondents to completely answer the question

- Provide extra motivation to respond

- Use scrollable boxes on internet surveys

---

### Explanatory

.pull-left[
<center>
<img src="img/image27a.png" width="59%" height="59%" />
</center>
]

.pull-right[
]

.pull-left[
]

.pull-right[
<center>
<img src="img/image27b.png" width="92%" height="92%" />
</center>
]

---

## Partially Closed

---

### Partially Closed Nominal

---

# Part II: The Difficulty in Designing Mixed Methods Surveys

---

## Notes

> Beneficial for resource constraint studies

> Extremely difficult to construct yielding a high cost-high reward scenario

> Multiple threats to internal validity exist including attrition, maturation, and selection

> There is a split view on the existence of *true* mixed surveys

---

## Remember Its Methods not Methodology

.pull-left[

Methods 
specific tools and procedures used to collect and analyze data

]

.pull-right[

Methodology 
the overarching strategy and rationale of an evaluation or research project

]

---

## Typically

.pull-left[
<center>
Easier: 
 
 
Triangulation
 
 
 
<img src="img/triangulation.png" width="60%" />
</center>
]

.pull-right[
<center>
Harder:
 
 
Sequential
 
 
 
<img src="img/explanatory.png" width="100%" />
 
 
 
<img src="img/exploratory.png" width="100%" />
</center>
]

---

## Triangulation Mixed Surveys

.pull-left[
 
<center>
<img src="img/triangulation-survey.png" width="60%" />
</center>
]

.pull-right[
 
Factors
> passive deciders. filter elements such as skip > logic, condition branching, etc.

> internal validity. prone to attrition and maturation threats

> psychological effects. subject to primary and recency effects due to same respondents
]

---

## Sequential Mixed Surveys

.pull-left[
 
<center>
<img src="img/sequential-survey.png" width="95%" />
</center>
]

.pull-right[
 
Factors
> active deciders: filter elements such as the application of machine learning models, real-time evaluations

> internal validity. prone to selection threats

> psychological effects. subject to carry-over effects when using same respondents
]

---

## In a Nutshell

.center2[
Only do this in a worst case scenario!
]

---

#  That’s It!

Any questions?

<center>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>
</center>